Skip to content

Epson V700 Scanner: Add plate detection and colony phenotyping pipeline - #75

Merged
lanery merged 28 commits into
stagingfrom
rl/v700-plate-detection
May 18, 2026
Merged

Epson V700 Scanner: Add plate detection and colony phenotyping pipeline#75
lanery merged 28 commits into
stagingfrom
rl/v700-plate-detection

Conversation

@lanery

@lanery lanery commented May 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds automatic plate detection (agar plates inside gold frames) and renames TIFFToJPEGConverterTiffProcessor with a cleaner API
  • Implements a full colony detection and phenotyping pipeline: margin crop → contrast optimization → Difference-of-Gaussians filtering → Otsu threshold → morphological cleanup → per-colony measurements in physical units (mm)
  • Exports colony data as CSV and draws labeled bounding-box overlays on the JPEG preview; accessible via new --detect-colonies CLI flag
  • Increases MAX_DIMENSION of the exported JPEG preview to 2000 px (up from 1000 px). This is needed so that colony bounding-box overlays and numeric labels remain legible in the preview image.

Test plan

  • Run data-hub epson-scanner <tiff> --detect-colonies on a scan with visible colonies and verify overlay + CSV output
  • Run without --detect-colonies to confirm the default path is unchanged
  • Verify tests pass (make check-all in lambda/)

Made with Cursor

lanery and others added 27 commits May 8, 2026 15:34
skimage.io plugin infrastructure is deprecated since 0.25 and will be
removed in 0.27. Use imageio.v3 directly for JPEG read/write.

Co-authored-by: Cursor <cursoragent@cursor.com>
…r to TiffProcessor

Detect gold 3D-printed frames around agar plates using HSV
thresholding, emit plate_count/plate_boxes in metadata, and draw
green overlay rectangles on the JPEG preview. Remove integration
tests that relied on example JPG files not tracked in the repo.

Co-authored-by: Cursor <cursoragent@cursor.com>
Make plate_boxes None-able so parse_metadata() omits plate fields
when detection hasn't run (instead of silently returning zeros),
simplify overlay drawing to fill-and-restore (drops ski.draw loop),
rename converter -> processor, and mark PlateBox as private.

Co-authored-by: Cursor <cursoragent@cursor.com>
- Make detect_plates side-effect-only (results in self.plate_boxes)
- Guard against redundant detection in export_jpg
- Fix _to_rgb_uint8 redundant float round-trip for RGBA inputs
- Use round() instead of int() in _derive_dpi for robustness
- Add logging in detect_plates and warning in parse_metadata
- Document non-overlapping assumption in _draw_plate_overlays

Co-authored-by: Cursor <cursoragent@cursor.com>
New colony_detection module that operates on individual plate crops:
crop margin, optimize contrast, detect colony presence, Gaussian smooth,
and Otsu threshold. Measures per-colony area, centroid, eccentricity,
and equivalent diameter. Wired into the Lambda pipeline (auto-runs on
each detected plate, results stored in run metadata) and the CLI
(--detect-colonies flag on epson-scanner command).

Co-authored-by: Cursor <cursoragent@cursor.com>
Draw cyan contour outlines on per-plate overlay JPEGs and export a
combined colony-properties CSV (via pandas). Wired into both the CLI
(--detect-colonies now emits overlay JPEGs + CSV) and the Lambda
pipeline (uploads overlays and CSV to S3 as processed files).

Co-authored-by: Cursor <cursoragent@cursor.com>
Instead of producing separate per-plate colony overlay images, colony
contours are now drawn directly on the original export figure alongside
the plate bounding boxes before resizing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the 10% fractional margin with a fixed 200px crop from each
edge, which better handles varying plate sizes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Have _draw_colony_contours import MARGIN_PX from colony_detection
instead of hardcoding its own default, so the crop offset stays in sync.

Co-authored-by: Cursor <cursoragent@cursor.com>
The 95th percentile was too sensitive, causing empty plates to be
misclassified as having colonies due to noise/edge artefacts.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ssians

Use a DoG band-pass filter (low_sigma=0.6, high_sigma=64) followed by
10th-percentile baseline subtraction for colony detection pre-processing.
This suppresses both high-frequency noise and low-frequency background
gradients more effectively than simple Gaussian smoothing.

Co-authored-by: Cursor <cursoragent@cursor.com>
- tweak DoG parameters
Test helpers defaulted to 400x400 images which became empty after
cropping 200px from each side. Increase defaults to 600x600 and add
an empty-array guard in detect_colony_presence.

Co-authored-by: Cursor <cursoragent@cursor.com>
- minor correction to docstring
- modify display parameters
Convert all spatial colony properties from pixels to millimetres using
scan DPI (25.4 / dpi). Centroid and bbox coordinates are now
plate-relative (margin offset added back) for downstream colony picking.
Per-colony mean RGB extracted via regionprops intensity_image.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Colonies that touch the edge of the margin-cropped region are partial
and have distorted properties. Also bump MARGIN_PX to 250 and increase
test plate sizes accordingly.

Co-authored-by: Cursor <cursoragent@cursor.com>
The JPG export draws contours from result.mask. Previously it used the
raw Otsu mask which still contained border-touching colonies that were
filtered from the properties list. Now measure_colonies returns the
post-clear_border mask so contours and properties stay consistent.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace pixel-level contour stamping with matplotlib-rendered bounding
boxes and numeric labels. Clean up inline imports (move MARGIN_PX to
top-level, use Figure/FigureCanvasAgg directly instead of pyplot with
global backend side effect) and replace BytesIO roundtrip with direct
canvas buffer read.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use skimage remove_small_objects to filter noise artifacts from the
binary mask before labeling. The threshold is converted from mm² to
pixels using the scan DPI.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
- label after clear border
Extract run_colony_pipeline() so both the Lambda handler and CLI
delegate to one function instead of duplicating the plate-loop logic.
Fix remove_small_objects deprecation (min_size → max_size) and a
test that used a blob below the minimum-area threshold.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vercel

vercel Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
data-hub Ready Ready Preview, Comment May 18, 2026 9:34pm

Request Review

@lanery
lanery requested a review from wasimxyz May 18, 2026 18:37

@wasimxyz wasimxyz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested changes locally with 20260511_3dptest_yeast001 and observed that the annotated JPG and CSV are generated successfully. Looks good to merge. Discussed offline:

  • The plate detection and colony phenotyping isn't always accurate, but we're okay to defer and make follow-up improvements.
  • I will add frontend changes in a follow up PR.

- Guard _to_rgb_uint8 against unintended rescale_intensity on RGBA uint8
  inputs by tracking the original dtype before rgba2rgb conversion.
- Cache _to_rgb_uint8 result on TiffProcessor to avoid redundant ~100 MB
  allocations across detect_plates/crop_plates/export_jpg.
- Make parse_metadata() call detect_plates() itself when plate_boxes is
  None, removing the order-dependent API.
- Fix variable shadowing in measure_colonies (labels → cleared_mask).
- Promote lazy colony_detection imports to top-level in process_file.py.
- Update module docstring to document all 6 pipeline steps.
- Add test for sub-threshold colony removal.
- Add smoke tests for _draw_plate_overlays and _draw_colony_bboxes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lanery
lanery merged commit 28e7c46 into staging May 18, 2026
4 checks passed
@lanery
lanery deleted the rl/v700-plate-detection branch May 18, 2026 21:38
wasimxyz added a commit that referenced this pull request May 19, 2026
…ve the JPG preview (#76)

Renders the `*_colonies.csv` produced by the Epson V700 colony detection
pipeline (PR #75) as a 10-rows-per-page table above the JPG in
RunReportSection. CSV bytes are fetched client-side via the existing
presigned-download redirect so they flow browser → S3 directly. Column
headers are derived from the CSV so the schema can evolve without UI changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants